Papers with Multimodal large language models
Fast or Slow? Integrating Fast Intuition and Deliberate Thinking for Enhancing Visual Question Answering (2025.acl-short)
Copied to clipboard
| Challenge: | Current approaches generate visual markers for all questions, generating excessive visual markers. |
| Approach: | They propose a plug-and-play approach that adapts to the complexity of questions . they propose combining fast intuitive judgments with deliberate analytical reasoning . |
| Outcome: | The proposed approach improves performance on four benchmarks on ScienceQA, TextQA, VizWiz, and MME. |
Cross-Modal Projection in Multimodal LLMs Doesn’t Really Project Visual Attributes to Textual Space (2024.acl-short)
Copied to clipboard
| Challenge: | Existing multimodal large language models are limited to general-purpose multimodal tasks like question-answering on natural images. |
| Approach: | They propose to use cross-modal projection networks and a large language model to model domain-specific visual attributes of MLLMs. |
| Outcome: | The proposed models gain domain-specific visual capabilities when the projection is fine-tuned, but the updates do not extract relevant domain-specific visual attributes. |
EFUF: Efficient Fine-Grained Unlearning Framework for Mitigating Hallucinations in Multimodal Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to eliminate hallucinations require expensive human annotation . hallucination in multimodal large language models poses unique challenges for current research . |
| Approach: | They propose a fine-grained unlearning framework that performs gradient ascent to eliminate hallucinations without paired data. |
| Outcome: | The proposed method reduces hallucinations while preserving quality with modest computational overhead. |
IPS: In-Prompt Process Supervision for Short Video Content Moderation (2026.acl-industry)
Copied to clipboard
| Challenge: | Multimodal large language models (MLLMs) capture semantics of short video content but fail to account for policy-specific details. |
| Approach: | They propose a framework that integrates In-prompt Process Supervision into MLLMs . they propose sequential reasoning over ancillary questions during fine-tuning . |
| Outcome: | IPS outperforms baseline MLLMs on public and proprietary benchmarks . replacing human-annotated ancillary labels with MLML-generated ones results in performance degradation. |
Multimodal Evaluation of Russian-language Architectures (2026.eacl-long)
Copied to clipboard
Artem Chervyakov, Ulyana Isaeva, Anton Emelyanov, Artem Safin, Maria Tikhonova, Alexander Kharitonov, Yulia Lyakh, Petr Surovtsev, Denis Shevelev, Vildan Saburov, Vasily Konovalov, Elisei Rykov, Ivan Sviridov, Amina Miftakhova, Ilseyar Alimova, Alexander Panchenko, Alexander Kapitanov, Alena Fenogenova
| Challenge: | Multimodal large language models (MLLMs) are at the center of research attention, yet intelligence, limitations, and risks remain insufficiently understood. |
| Approach: | They propose an open multimodal evaluation framework for Russian-spoken architectures . the framework is instruction-based and includes 18 newly constructed evaluation tasks . |
| Outcome: | The proposed framework provides a replicable methodology for constructing multimodal benchmarks in Russian-spoken architectures. |
MTMCS-Bench: Evaluating Contextual Safety of Multimodal Large Language Models in Multi-Turn Dialogues (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing contextual safety benchmarks are mostly single-turn and miss how malicious intent can emerge gradually or how the same scene can support both benign and exploitative goals. |
| Approach: | They propose a benchmark that evaluates contextual safety in multimodal large language models . they observe persistent trade-offs between contextual safety and utility . |
| Outcome: | The proposed model combines multi-turn and multi-switch scenarios to evaluate safety in multimodal large language models. |
SPARTA: Evaluating Reasoning Segmentation Robustness through Black-Box Adversarial Paraphrasing in Text Autoencoder Latent Space (2026.eacl-long)
Copied to clipboard
Viktoriia Zinkovich, Anton Antonov, Andrei Spiridonov, Denis Shepelev, Andrey Moskalenko, Daria Pugacheva, Elena Tutubalina, Andrey Kuznetsov, Vlad Shakhuro
| Challenge: | Existing work on semantically equivalent textual paraphrases has focused on perturbing image inputs. |
| Approach: | They propose a novel adversarial paraphrasing task that generates grammatically correct paraphrases that sighed the original query meaning while degrading segmentation performance. |
| Outcome: | The proposed task outperforms previous methods by up to 2x on ReasonSeg and LLMSeg-40k datasets. |
Shallow Focus, Deep Fixes: Enhancing Shallow Layers Vision Attention Sinks to Alleviate Hallucination in LVLMs (2025.emnlp-main)
Copied to clipboard
Xiaofeng Zhang, Yihao Quan, Chen Shen, Chaochen Gu, Xiaosong Yuan, Shaotian Yan, Jiawei Cao, Hao Cheng, Kaijie Wu, Jieping Ye
| Challenge: | Multimodal large language models (MLLMs) demonstrate excellent abilities for understanding visual information, but the hallucination remains a challenging problem. |
| Approach: | They propose a training-free approach to enhance vision attention sinks to facilitate convergence of the image token attention sink within shallow layers. |
| Outcome: | The proposed approach improves the convergence of the image token attention sink within shallow layers and strengthens the layer’s focus on the image itself. |
Unveiling Uncertainty: A Deep Dive into Calibration and Performance of Multimodal Large Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | Multimodal large language models combine visual and textual data for tasks like image captioning and visual question answering. |
| Approach: | They propose temperature scaling and iterative prompt optimization to calibrate MLLMs and enhance model reliability. |
| Outcome: | The proposed techniques improve MLLMs and improve model reliability. |
MLLM-Bench: Evaluating Multimodal LLMs with Per-sample Criteria (2025.naacl-long)
Copied to clipboard
Wentao Ge, Shunian Chen, Hardy Chen, Nuo Chen, Junying Chen, Zhihong Chen, Wenya Xie, Shuo Yan, ChenghaoZhu ChenghaoZhu, Ziyue Lin, Dingjie Song, Xidong Wang, Anningzhe Gao, Zhang Zhiyi, Jianquan Li, Xiang Wan, Benyou Wang
| Challenge: | Existing evaluation methodologies for multimodal large language models are limited in evaluating objective queries without considering real-world user experiences. |
| Approach: | They propose to evaluate multimodal large language models with per-sample criteria using potent MLLM as the judge. |
| Outcome: | The proposed evaluation paradigm shows that it can be used to evaluate multimodal large language models with per-sample criteria. |
Multimodal Language Models See Better When They Look Shallower (2025.emnlp-main)
Copied to clipboard
Haoran Chen, Junyan Lin, Xinghao Chen, Yue Fan, Jianfeng Dong, Xin Jin, Hui Su, Jinlan Fu, Xiaoyu Shen
| Challenge: | Existing studies show that multimodal large language models extract visual features from the final layers of a pretrained Vision Transformer. |
| Approach: | They propose a feature fusion method that strategically incorporates shallower layers . they propose MLLMs that extract visual features from the final layers of a pretrained Vision Transformer . |
| Outcome: | The proposed method outperforms deep layers on fine-grained visual tasks . it is the first comprehensive study of visual layer selection for MLLMs . |
AXIS: Efficient Human-Agent-Computer Interaction with API-First LLM-Based Agents (2025.acl-long)
Copied to clipboard
Junting Lu, Zhiyang Zhang, Fangkai Yang, Jue Zhang, Lu Wang, Chao Du, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, Qi Zhang
| Challenge: | Multimodal large language models (MLLMs) have enabled LLM-based agents to directly interact with application user interfaces (UIs), enhancing agents’ performance in complex tasks. |
| Approach: | They propose a novel agent framework that prioritizes actions through application programming interfaces over UI actions and facilitates the creation and expansion of APIs through automated exploration of applications. |
| Outcome: | The proposed framework reduces task completion time by 65%-70% and cognitive workload by 38%-53% while maintaining accuracy of 97%-98% compared to humans. |
SURE: Safety Understanding and Reasoning Enhancement for Multimodal Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing multimodal large language models incorporate visual and textual information, but introduces new and complex safety risks. |
| Approach: | They propose a safety reasoning framework that integrates visual modalities into multimodal models to help them resist jailbreak attacks. |
| Outcome: | The proposed framework improves model safety while avoiding over-defense . it is based on a large-scale safety reasoning dataset . |
VQAGuider: Guiding Multimodal Large Language Models to Answer Complex Video Questions (2025.acl-long)
Copied to clipboard
| Challenge: | Multimodal large language models (MLLMs) can grasp the intention of a question and decomposing it to a series of visual recognition sub-tasks to find out the answer with the help of an agent. |
| Approach: | They propose a framework for multimodal large language models to grasp the intention of a question and decompose it into a series of visual recognition sub-tasks to find out the answer. |
| Outcome: | The proposed framework improves the accuracy of complex video-related questions by 29.6% and 17.2% on CVQA and the existing VQA datasets. |
VLSBench: Unveiling Visual Leakage in Multimodal Safety (2025.acl-long)
Copied to clipboard
| Challenge: | Existing studies show that textual unlearning does not achieve comparable safety performance with image-text alignment. |
| Approach: | They propose to use textual unlearning to align MLLMs with image-text pairs to explain this problem . they construct a visual leakless safety bench with 2.2k image- text pairs to test this problem. |
| Outcome: | The proposed model can refuse image-text pairs according to textual queries, leading to unreliable safety evaluations. |
VFA: Empowering Multilingual MLLMs via Vision-Free Adaptation (2026.acl-long)
Copied to clipboard
Yixia Li, Yaqing Shi, Zhiwen Ruan, Dongdong Zhang, Lingjie Jiang, Shaohan Huang, Yun Chen, Guanhua Chen, Furu Wei
| Challenge: | Multimodal large language models have advanced rapidly, yet most remain English-centric . scaling multilingual multimodal instruction tuning is limited by the scarcity and high cost of non-English image–text supervision. |
| Approach: | They propose a framework that decouples multilingual language enhancement from visual alignment by composing complementary task vectors over a shared LLM backbone. |
| Outcome: | The proposed framework achieves competitive performance with a fully multimodally trained model using less than 2% of the text data. |
AHEAD: Attention Head Energy-Aware Dynamics for Hallucination Mitigation in MLLMs (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to hallucination mitigation ignore heterogeneous behaviors of attention heads . hallucinosity is a critical barrier to multimodal large language models' reliability, authors say . |
| Approach: | They propose a framework that quantifies the energetic properties of each attention head during object generation through two potential networks and dynamically adjusts their contributions at inference time. |
| Outcome: | The proposed framework reduces hallucination rates without fine-tuning the base model while maintaining generation quality. |
RRHF-V: Ranking Responses to Mitigate Hallucinations in Multimodal Large Language Models with Human Feedback (2025.coling-main)
Copied to clipboard
| Challenge: | Existing methods to mitigate hallucinations generate erroneous or fabricated information. |
| Approach: | They propose a rank-response-based model that annotates pair-reponses and trains alignment algorithms to improve the correspondence between images and text. |
| Outcome: | The proposed model outperforms the DPO method and outperfies existing methods on two MLLMs of different sizes and four widely used benchmarks. |
CMIE: Combining MLLM Insights with External Evidence for Explainable Out-of-Context Misinformation Detection (2025.findings-acl)
Copied to clipboard
| Challenge: | Multimodal large language models have demonstrated impressive capabilities in visual reasoning and text generation. |
| Approach: | They propose a multimodal large language model that captures deeper relationships between images and text . they propose CMIE, which uses a Coexistence Relationship Generation strategy and an AS mechanism to detect misinformation. |
| Outcome: | The proposed framework outperforms existing methods in detecting out-of-context misinformation. |
Visual Self-Fulfilling Alignment: Shaping Safety-Oriented Personas via Threat-Related Images (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods require explicit safety labels or contrastive data, yet visual inputs enable harmful outputs. |
| Approach: | They propose a visual self-fulfilling alignment mechanism that fine-tunes vision-language models on neutral VQA tasks without any safety labels. |
| Outcome: | The proposed approach reduces attack success rate, improves response quality, and mitigates over-refusal while preserving general capabilities. |
CoCo-Agent: A Comprehensive Cognitive MLLM Agent for Smartphone GUI Automation (2024.findings-acl)
Copied to clipboard
| Challenge: | Current vital challenges for autonomous agents lie in two aspects: dependence on strong (M)LLMs and insufficient GUI environment modeling. |
| Approach: | They propose a comprehensive cognitive LLM agent with two novel approaches to improve GUI automation performance. |
| Outcome: | The proposed agent achieves state-of-the-art performance on AITW and META-GUI benchmarks. |
CODIS: Benchmarking Context-dependent Visual Comprehension for Multimodal Large Language Models (2024.acl-long)
Copied to clipboard
Fuwen Luo, Chi Chen, Zihao Wan, Zhaolu Kang, Qidong Yan, Yingjie Li, Xiaolong Wang, Siyu Wang, Ziyue Wang, Xiaoyue Mi, Peng Li, Ning Ma, Maosong Sun, Yang Liu
| Challenge: | Multimodal large language models have demonstrated promising results in a variety of tasks that combine vision and language. |
| Approach: | They propose a benchmark to assess the ability of models to use contextual information in free-form text to enhance visual comprehension. |
| Outcome: | The proposed model fails to extract and utilize contextual information to improve understanding of images. |
TokenPenalty: Alleviating Attention Sinks and Positional Decay in LVLMs (2026.findings-acl)
Copied to clipboard
| Challenge: | Multimodal large language models (MLLMs) often hallucinate due to two relevant phenomena: massive activation phenomenon and positional information decay. |
| Approach: | They propose a token-level intervention strategy that dynamically suppresses irrelevant visual tokens while preserving key contextual signals. |
| Outcome: | Experiments show that TokenTruth significantly improves factual consistency across MLLMs on standard image understanding benchmarks. |
Peacock: A Family of Arabic Multimodal Large Language Models and Benchmarks (2024.acl-long)
Copied to clipboard
Fakhraddin Alwajih, El Moatez Billah Nagoudi, Gagan Bhatia, Abdelrahman Mohamed, Muhammad Abdul-Mageed
| Challenge: | MLLMs have proven effective in a wide range of tasks that require complex reasoning and linguistic comprehension, but they are limited to English-based settings. |
| Approach: | They propose a family of Arabic multimodal large language models with strong vision and language capabilities. |
| Outcome: | The proposed models show strong performance on visual reasoning tasks and language capabilities. |
SceneAlign: Aligning Multimodal Reasoning to Scene Graphs in Complex Visual Scenes (2026.acl-long)
Copied to clipboard
Chuhan Wang, Xintong Li, Jennifer Yuntong Zhang, Junda Wu, Chengkai Huang, Lina Yao, Julian McAuley, Jingbo Shang
| Challenge: | Existing preference-based approaches fail to address this challenge by exploiting language priors to bypass visual grounding. |
| Approach: | They propose a framework that leverages scene graphs as structured visual information to perform controllable structural interventions. |
| Outcome: | The proposed framework improves answer accuracy and reasoning faithfulness across seven visual reasoning benchmarks. |
Token Pruning in Multimodal Large Language Models: Are We Solving the Right Problem? (2025.findings-acl)
Copied to clipboard
| Challenge: | Multimodal large language models have shown remarkable performance for cross-modal understanding and generation, yet suffer from severe inference costs. |
| Approach: | They propose to prune redundant tokens in MLLMs to reduce computation and storage costs. |
| Outcome: | The proposed method reduces the computational and storage costs of MLLMs by identifying redundant tokens and pruning them. |
MC-MKE: A Fine-Grained Multimodal Knowledge Editing Benchmark Emphasizing Modality Consistency (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks for knowledge editing in multimodal large language models focus on limited scenarios due to the lack of rigorous definition of multimodal knowledge. |
| Approach: | They propose a decomposed definition of multimodal knowledge and a benchmark to evaluate it. |
| Outcome: | The proposed method reveals that it is difficult to define multimodal knowledge editing in LLMs. |
Self-adaptive Dataset Construction for Real-World Multimodal Safety Scenarios (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing dataset construction methods fail to cover the complexity of multimodal safety scenarios . lack of a unified evaluation metric makes them unproven . |
| Approach: | They propose a risk-oriented image-oriented self-adaptive dataset construction method for RMS . they automatically generate an RMS dataset comprising 35,610 image–text pairs with guidance responses . |
| Outcome: | The proposed method automatically generates an RMS dataset comprising 35,610 image–text pairs with guidance responses. |
SafeSteer: A Decoding-level Defense Mechanism for Multimodal Large Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing defense methods rely on fine-tuning or inefficient post-hoc interventions, limiting their ability to address novel attacks. |
| Approach: | They propose a decoding-level defense mechanism that employs a lightweight discriminator to iteratively steer the decoding process toward safety. |
| Outcome: | The proposed method improves safety performance by up to 33.40% without fine-tuning on multiple MLLMs. |
Exploring Response Uncertainty in MLLMs: An Empirical Evaluation under Misleading Scenarios (2025.emnlp-main)
Copied to clipboard
Yunkai Dang, Mengxi Gao, Yibo Yan, Xin Zou, Yanggan Gu, Jungang Li, Jingyu Wang, Peijie Jiang, Aiwei Liu, Jia Liu, Xuming Hu
| Challenge: | Existing studies have focused mainly on visual–textual misalignment, leaving largely unexplored the MLLMs’ ability to preserve an original correct answer when confronted with misleading information. |
| Approach: | They propose a two-stage evaluation pipeline to quantify the response uncertainty phenomenon by eliciting each model’s original response on unperturbed inputs and injecting explicit (false-answer hints) and implicit (contextual contradictions) misleading instructions. |
| Outcome: | The proposed model overturns a correct answer in 65% of cases after receiving a single deceptive cue. |
DRIFT: Transferring Reasoning Priors for Efficient MLLM Fine-Tuning (2026.findings-acl)
Copied to clipboard
Chao Huang, Zeliang Zhang, Jiang Liu, Ximeng Sun, Jialian Wu, Xiaodong Yu, Ze Wang, Chenliang Xu, Emad Barsoum, Zicheng Liu
| Challenge: | Multimodal large language models (MLLMs) have made rapid progress in perception and alignment, but their reasoning ability often lags behind strong text-only LLMs. |
| Approach: | They propose a method that transfers reasoning knowledge in the gradient space while preserving multimodal alignment. |
| Outcome: | Experiments on multimodal reasoning benchmarks show that DRIFT outperforms naive merging and standard SFT. |
Pointing to a Llama and Call it a Camel: On the Sycophancy of Multimodal Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Multimodal large language models exhibit a pronounced form of visual sycophantic behavior when they process image inputs. |
| Approach: | They propose a technique that allows multimodal large language models to engage in reflective reasoning and determine whether a user’s instruction is misleading or corrective. |
| Outcome: | The proposed model resists misleading instructions but is stubborn even if it is wrong. |
InImageTrans: Multimodal LLM-based Text Image Machine Translation (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing multimodal large language models suffer from repetition and omission hallucinations when transferred to text image machine translation task. |
| Approach: | They propose an efficient MLLM named InImageTrans for TiMT and a method for advancing it. |
| Outcome: | The proposed method outperforms existing open-source MLLMs on the MCiT benchmark. |
Multimodal Language Models Show Evidence of Embodied Simulation (2024.lrec-main)
Copied to clipboard
| Challenge: | Multimodal large language models (MLLMs) are gaining popularity as partial solutions to the “symbol grounding problem” faced by language models trained on text alone. |
| Approach: | They propose to use multimodal large language models to integrate linguistic representations with data from other modalities to investigate whether they are integrated into a model. |
| Outcome: | The proposed models are sensitive to visual features like object shape when it is implied by a verbal description of an event. |
Decoupled Proxy Alignment: Mitigating Language Prior Conflict for Multimodal Alignment in MLLMs (2025.findings-emnlp)
Copied to clipboard
Chenkun Tan, Pengyu Wang, Shaojun Zhou, Botian Jiang, Zhaowei Li, Dong Zhang, Xinghao Wang, Yaqian Zhou, Xipeng Qiu
| Challenge: | Recent advances in multimodal large language models focus on improving performance . however, language prior conflict leads to suboptimal vision-language alignment . |
| Approach: | They propose a method to decouple the alignment process from language prior interference . they use a proxy LLM to detach from language interference during pretraining . |
| Outcome: | The proposed method improves training performance and generalizes training data. |
MEDSYN: Benchmarking Multi-EviDence SYNthesis in Complex Clinical Cases for Multimodal Large Language Models (2026.findings-acl)
Copied to clipboard
Boqi Chen, Xudong Liu, Jiachuan Peng, Marianne Frey-Marti, Kyle Lam, Bang Zheng, Lin Li, Jianing Qiu
| Challenge: | Existing benchmarks for multimodal large language models do not capture real-world clinical complexity. |
| Approach: | They evaluate multilingual, multimodal multimodal models of clinical cases with up to 7 distinct visual clinical evidence types per case. |
| Outcome: | The proposed model outperforms human models on differential diagnosis (DDx) generation and final diagnosis (FDx) selection. |
VisFinEval: A Scenario-Driven Chinese Multimodal Benchmark for Holistic Financial Understanding (2025.emnlp-main)
Copied to clipboard
Zhaowei Liu, Xin Guo, Haotian Xia, Lingfeng Zeng, Fangqi Lou, Jinyi Niu, Mengping Li, Qi Qi, Jiahuan Li, Wei Zhang, Yinglong Wang, Weige Cai, Weining Shen, Liwen Zhang
| Challenge: | Existing benchmarks focus on text comprehension, but MLLMs lack the ability to integrate visual data over financial visuals. |
| Approach: | They evaluate 21 state-of-the-art multimodal large language models in a zero-shot setting . they use an annotated question–answer pair from eight common financial image modalities . |
| Outcome: | The new benchmark outperforms existing models but trailed financial experts by 14 percentage points. |
Look and Think: Efficient Multimodal Reasoning via Modality-Decoupled Compression (2026.findings-acl)
Copied to clipboard
| Challenge: | Multimodal large language models have strong performance on visual question answering benchmarks . however, their inference efficiency is severely constrained by the rapidly growing context . |
| Approach: | They propose a modality-decoupled compression method that enables efficient multimodal inference . they propose to evict visual tokens whenever visual grounding is unnecessary . |
| Outcome: | The proposed method reduces the average context length by up to 57% while maintaining comparable performance to the standard MLLM baseline. |
MERMAID: Multi-perspective Self-reflective Agents with Generative Augmentation for Emotion Recognition (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing multimodal large language models struggle to handle ambiguous emotional expressions and implicit affective cues, which are crucial for affective understanding but largely overlooked. |
| Approach: | They propose a multi-agent framework that integrates a self-reflection module, an emotion-guided visual augmentation module, and a cross-modal verification module to enhance emotion recognition. |
| Outcome: | Extensive experiments show that MERMAID outperforms existing methods and achieves absolute accuracy gains of 8.70%–27.90% across diverse benchmarks. |
Hidden in Plain Sight: Reasoning in Underspecified and Misspecified Scenarios for Multimodal LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Multimodal large language models are increasingly deployed in open-ended, real-world environments where inputs are messy, underspecified, and not always trustworthy. |
| Approach: | They evaluate multimodal large language models in real-world environments where inputs are messy, underspecified, and not always trustworthy. |
| Outcome: | The proposed models fail to detect hidden issues even when they possess the necessary perceptual and reasoning skills. |
When Seeing Is not Enough: Revealing the Limits of Active Reasoning in MLLMs (2026.acl-long)
Copied to clipboard
| Challenge: | Existing evaluations of multimodal large language models focus on passive inference, where seeing is not enough. |
| Approach: | They propose a benchmark to evaluate active reasoning in multimodal large language models . they propose to acquire missing evidence and iteratively refine decisions under incomplete information . |
| Outcome: | The proposed model performs better on active reasoning than on passive inference settings. |
Understanding GUI Agent Localization Biases through Logit Sharpness (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Multimodal large language models often exhibit hallucinations that compromise reliability . despite promising performance, these models often display systematic localization errors . |
| Approach: | They propose a framework that categorizes model predictions into four distinct types . they propose metric that evaluates alignment between semantic continuity and logits distribution . |
| Outcome: | The proposed framework categorizes model predictions into four different types . it reveals nuanced failure modes beyond traditional accuracy metrics . |
AutoRubric: Rubric-Based Generative Rewards for Faithful Multimodal Reasoning (2026.findings-acl)
Copied to clipboard
| Challenge: | Multimodal large language models (MLLMs) have advanced from perception tasks to complex multi-step reasoning. |
| Approach: | They propose a framework that integrates reinforcement learning with verifiable rewards with process-level supervision through automatically collected rubric-based generative rewards. |
| Outcome: | The proposed framework achieves state-of-the-art performance on six multimodal reasoning benchmarks and significantly improves reasoning faithfulness in dedicated evaluations. |
Beyond the Panorama: Training-Free Hierarchical Perception-Reasoning for Fine-Grained Vision in MLLMs (2026.acl-long)
Copied to clipboard
| Challenge: | Existing multimodal large language models (MLLMs) face challenges in fine-grained visual tasks. |
| Approach: | They propose a training-free hierarchical perception-reasoning framework that enhances fine-grained visual understanding by simulating human perception mechanisms. |
| Outcome: | The proposed framework enhances fine-grained visual understanding by simulating human perception mechanisms. |
Reducing Peak Memory Usage for Modern Multimodal Large Language Model Pipelines (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to reduce memory usage of multimodal large language models rely on storing large numbers of vision tokens in the key–value cache . however, such compression is typically only applied after all inputs are processed, resulting in high peak memory usage during the prefill stage. |
| Approach: | They propose a sequential input-compression mechanism that enforces a fixed memory budget by performing structure-aware key–value cache compression during the prefill stage. |
| Outcome: | The proposed approach reduces peak memory usage while maintaining generative performance with only minimal degradation, enabling more practical and memory-efficient multimodal inference. |
Is Cognition Consistent with Perception? Assessing and Mitigating Multimodal Knowledge Conflicts in Document Understanding (2025.emnlp-main)
Copied to clipboard
| Challenge: | Multimodal large language models (MLLMs) have shown impressive capabilities in document understanding due to different types of annotation noise in training. |
| Approach: | They propose a method to reduce C&P knowledge conflicts across all tested MLLMs . they propose to use annotation noise to train models to understand document content . |
| Outcome: | The proposed method reduces C&P knowledge conflicts across all tested MLLMs and enhances their performance in both cognitive and perceptual tasks. |
Taking Notes Brings Focus? Towards Multi-Turn Multimodal Dialogue Learning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing multimodal large language models are trained on single-turn vision question-answering tasks, which do not accurately reflect real-world human conversations. |
| Approach: | They propose a large-scale multi-turn multimodal dialogue dataset that uses rules and GPT assistance to generate a multi-turned multimodal dialog dataset. |
| Outcome: | The proposed dataset is a strong benchmark for multi-turn multimodal dialogue learning . it features complex dialogues with contextual dependencies that force models to track, ground, and recall information across multiple turns and disparate visual regions. |
CCD: Mitigating Hallucinations in Radiology MLLMs via Clinical Contrastive Decoding (2026.findings-acl)
Copied to clipboard
| Challenge: | Multimodal large language models generate medical hallucinations due to over-sensitivity to clinical sections. |
| Approach: | They propose a framework that integrates structured clinical signals from task-specific radiology expert models. |
| Outcome: | The proposed framework improves overall performance on radiology report generation (RRG) on the MIMIC-CXR dataset, it yields up to 17% improvement in RadGraph-F1. |
MirrorQA: Benchmarking Multimodal LLMs on Mirror-Orientation Reasoning (2026.acl-long)
Copied to clipboard
| Challenge: | Multimodal large language models (MLLMs) have achieved remarkable progress in recent years, yet their ability to perform left–right reasoning in mirror contexts remains underexplored. |
| Approach: | They propose a benchmark to evaluate MLLMs' ability to distinguish left from right from a subject-centered perspective. |
| Outcome: | The proposed benchmarks show that even the best performing models achieve only 65.40% accuracy, far below the 99.28% accuracy of humans. |
Revealing the Seen, Imagining the Beyond: A Survey of Image-Grounded Chain-of-Thought Reasoning in Multimodal LLMs (2026.acl-long)
Copied to clipboard
| Challenge: | Recent advances in Multimodal Large Language Models (MLLMs) have shifted visual reasoning from tool-calling to end-to-end perceptionreasoning. |
| Approach: | They synthesize the emerging paradigm of Image-Grounded Chain-of-Thought (IG-CoT) they propose a method-centric taxonomy covering prompting, supervised fine-tuning, and reinforcement learning . |
| Outcome: | The proposed model is based on a method-centric taxonomy and benchmarks. |
Mind’s Eye: A Benchmark of Visual Abstraction, Transformation and Composition for Multimodal LLMs (2026.acl-long)
Copied to clipboard
| Challenge: | Existing evaluations of multimodal large language models (MLLMs) have demonstrated compelling visual understanding in recent years. |
| Approach: | They propose a multimodal large language model with eight visuo-cognitive tasks inspired by classic human intelligence tests organized under a novel A–R–T taxonomy: Abstraction, Relation, and Transformation. |
| Outcome: | The proposed frameworks are based on eight visuo-cognitive tasks inspired by human intelligence tests and organized under a novel A–R–T taxonomy: Abstraction, Relation, and Transformation. |